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EXECUTIVE SUMMARY 


Overview of School Network Learning Project 

In 2007, the Columbia University Middle East Research Center (CUMERC), with the 
support and patronage of Her Majesty Queen Rania Al Abdullah, facilitated the 
founding of the Queen Rania Teacher Academy (QRTA) to help advance education 
in Jordan and throughout the Middle East. Through CUMERC, QRTA and Teachers 
College, Columbia University (TC) formed a new partnership whose goal was to 

use high quality in-service training to improve the quality of the public schools in 
Jordan. The Consortium for Policy in Education at TC (CPRE) took on this work 
and began collaborating with QRTA to engage current educators in the adoption 
and use of evidence-based instructional practices in Jordan’s elementary and 
secondary schools. This initiative became known as the School Networks Learning 
Project and supported networks of schools in different regions of the country 

as vehicles for providing professional development of teachers in English, 
mathematics, and science as well as leadership training for principals and education 
supervisors to support the desired changes in classroom practice. 


Five core practices were emphasized during the professional development with the 
goal being that teachers would focus on these practices and take them back to their 
classrooms. Schools and teachers made a 2 to 3 year commitment to the project. 

At the time of the writing of this report, three cohorts have completed the Project 
and two additional cohorts have begun the Project. Across the three completed 
cohorts, 2,158 teachers, 894 school leaders, and 104 other educators participated in 
the School Network Project, totaling 3,130 participants over the almost six years of 
the Project’s implementation. 


Purpose of the Report 

This report is the second evaluation report on the Project and focuses on change 
over time in classroom practice, transfer of capacity, and sustainability. The 
research questions, below, were central to each of these concepts and guided 

the evaluation. 


Change in practice: To what extent has the program resulted in changes in 
teachers’ and administrators’ values, dispositions, and practices? (The focus 
is particularly on the third cohort for this question.) 


Transfer of capacity: To what extent has QRTA developed the capacity to 
lead, support, and develop instructional improvement efforts? 


Sustainability: How and to what extent have the schools been able and 
willing to sustain their instructional improvement efforts? What obstacles 
have they encountered in their efforts to implement and sustain the 

new practices? 


Multiple sources of data were used to explore theses questions with much of the 
findings in the report coming from the analysis of participant surveys and video 
observations of a sample of Cohort 3 teachers. 
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Key Findings 


Change in practice 
Videotaping of classroom observations was used to measure change in practice by 
participant teachers over time. 


m There was positive change among the participants on over 80% of 
the measured indicators. 


m The greatest change occurred in classroom environment and 
lesson development. 


m= Female participants demonstrated greater changes than their male peers but it is 
important to note that the majority of the participants were women. 


Content knowledge remained a growth area for the majority of the participants, in 
Cohort 3 as reported by teacher and school leader participant surveys and analysis 
of the videos. 


Participant surveys also indicated positive change in practices both among teachers and 
school leaders. 


m The majority of teachers in Cohort 3, across the networks, reported changes in 
how often they used student collaboration and engagement in the classroom. 


m Teachers reported an increase in how often they encouraged student 
discussion in class. 


= More than 80% of the math and English writing teachers reported increases in 
their use of mixed groups or pairs; the same for over 70% of science teachers. 


m School leaders reported observing teachers more often. 


m School leaders noticed positive changes in the types of questions teachers asked 
in the classroom noting that they were higher level questions and that they 
probed for student understanding. 


Challenges still remain for both teachers and school leaders that will need to be addressed 
in the future. 


m While teachers are able to collaborate more often, the amount time spent in 
these meetings is still limited. 


m The majority of teachers and school leaders reported concerns about teachers’ 
content knowledge and their confidence in their understanding of the content. 


Transfer of capacity 
Overall, collaboration between the CPRE team and QRTA team members led to a 
successful transfer of knowledge and skill to QRTA staff. 


m Participants’ reports indicated that the workshops’ organization, content 
presented, and usefulness remained consistent, on average, across all three 
cohorts as the transfer to QRTA took place. 


= The number of QRTA staff contributing to the Project has almost tripled 
growing from only 9 team members in 2009 to 25 team members in 2014. 


Sustainability 
While not perfect, systems are in place to build the capacity of the network communities 
to sustain both the Project and the learning communities that develop from the program. 


= After the first year of the project, on-site support at schools was provided 
somewhat unevenly by QRTA. 


m QRTA staff believe on-site support is important and are making efforts 
to meet participants needs. 


m The majority of the teacher networks in Cohort 2 and Cohort 3 held 
the four required network meeting with many holding additional meetings. 


Conclusions 

Overall, the School Network Project has had a positive impact on both educators 
and the staff at the Queen Rania Teacher Academy. Continuing to engage the 
program participants and supporting them in their schools is an on-going effort. 
QRTA has acknowledged the challenges that the partnership faced during the first 
five years of implementation and is exploring ways to improve the program. 
Adapting to the needs of the participants has been a critical part of the program 
and continuing to assess their needs and how to meet them will continue to be a 
key component of the program’s success. 


CHANGING CLASSROOM PRACTICE: The Evaluation of the School Network Project in Jordan 


OVERVIEW OF SCHOOL NETWORK PROJECT 


In 2007, the Columbia University Middle East Research Center (CUMERC), with 
the support and patronage of Her Majesty Queen Rania Al Abdullah, facilitated the 
founding of the Queen Rania Teacher Academy (QRTA) to help advance education 
in Jordan and throughout the Middle East. Through CUMERC, the newly created 
Queen Rania Teacher Academy (QRTA) and Teachers College, Columbia University 
(TC) formed a new partnership. The goal of this partnership was to improve the 
quality of the public education system through the provision of high quality 
in-service training for current teachers. The Consortium for Policy in Education 

at TC (CPRE) took on this work and began collaborating with QRTA. Funded by 
the Jordan Ministry of Planning and United States Agency for International 
Development (USAID), and supported by the Ministry of Education (MOE), the 
partnership developed a model of high quality professional development that was 
scalable within Jordan. It supports the adoption and use of evidence-based 
instructional practices in Jordan’s elementary and secondary schools. This initiative 
became known as the School Networks Learning Project, also referred to as “the 
Project” or “the program.” The Project created and supported networks of schools 
in different regions of the country as vehicles for providing professional 
development for teams of teachers in English, mathematics, and science as well as 
leadership training for principals and education supervisors to support the desired 
changes in classroom practice. Each network focused on improving instruction in 
one of the three content areas: English, mathematics, or science. The first cohort 
of participants started the Project in 2009 with four additional cohorts beginning 
in the subsequent years. 


Goals of the School Network Learning Project 

The plan for instructional improvement was guided by research about 

(1) comprehensive approaches to school improvement, (2) effective professional 
development practices, and (3) effective instructional practices. The professional 
development workshops led by skilled trainers formed the foundation of the 
project. The plan was to provide nine events over three years to develop 
momentum for the changes and sustain the engagement to support 
implementation. The design and the delivery of the professional development 
were guided by evidence that effective programs include opportunities for active, 
hands-on learning, provide sustained and intensive support to deepen learning 
and use of the new strategies and content, utilize experienced practitioners as 
trainers, show respect for teacher knowledge and experience, and allow time for 
reflection and feedback from participants. The focus and content of the professional 
development were guided by substantive research that has identified some of the 
most effective instructional practices and strategies while incorporating the needs 
of the educators in the program. 
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Five core practices were emphasized during the professional development with the 
goal being that teachers would focus on these practices and take them back to their 
classrooms. The core practices, which are often referred to at the “big effects” or 
“high-leverage” practices in the workshop context, are: 


1) Lesson design: connecting the lesson to important concepts or skills 
(standards), selecting appropriate tasks, opening the lesson and reviewing prior 
knowledge, engaging students, making learning goals explicit, providing time 
for students to explore the content and apply it, and closing the lesson with 
review and summary; 


2) Team-based instruction: using various forms of peer and group learning or 
teaming with careful attention given to how class size, group composition and 
roles, seating arrangements, group stability, the number of groups, and other 
factors influence the effectiveness of this approach; 


3) Academically focused and rigorous tasks: Lessons should be built around 
tasks that are intellectually challenging (that is, have high cognitive demand), 
but also be achievable by all or almost all of the students in the class. Lessons 
that require students to communicate their understanding of the concepts 
and to explain the connections among concepts or to apply concepts to new 
problems have higher cognitive demand than lessons that ask students to 
memorize material or follow procedures or algorithms; 


4) Formative assessment/adaptive instruction: process of using data 
(observations, questioning, conferencing, quizzes, etc.) about student learning 
to adapt instruction to more effectively support student learning; and 


5) Student-centered discussion: student discourse in the classrooms that allows 
the teacher to understand their students’ academic thinking, and that allows 
students to develop greater understanding. 


Later in the project, a sixth practice, frequent writing in-class or out-of-class, was 
added to this list of high-leverage practices. 


Work at the schools—network meetings and school site visits—was intended to 
support teachers and leaders as they implemented the new practices, and to foster 
communities of practice that, once established, could sustain and extend the 
instructional improvement initiatives teachers and leaders were undertaking. 
Additionally, the hope was that these communities could cultivate a mutually 
reinforcing ethos of continuous instructional improvement within the school 
communities and within the content-area networks of schools. 


EVALUATING THE PROJECT 


This report is the second evaluation report in the second phase of the Project’s 
evaluation. For the first phase evaluation and report! a descriptive and formative 
evaluation approach was taken. In the second phase CPRE’s researchers shifted to 
an approach that attempts to measure change over time. Specifically, the research 
focused on three concepts: change, transfer, and sustainability. The following 
questions guided this phase of the evaluation: 


1) Change: To what extent has the program resulted in changes in teachers’ 
and administrators’ values, dispositions, and practices? (The focus is 
particularly on the third cohort for this question.) 


2) Transfer: To what extent has QRTA developed the capacity to lead, 
support, and develop instructional improvement efforts? 


a. How has their role developed, and what is the quality of the support 
that they have been provided by the project? 


b. How effective have the mechanisms for transferring leadership of 
the work to QRTA been? 


c. How does the quality of the Jordanian work compare to the work 
done by the CPRE teams? 


3) Sustainability: How and to what extent have the schools been able and 
willing to sustain their instructional improvement efforts? What obstacles 
have they encountered in their efforts to implement and sustain the 
new practices? 


! Copies of “Growing School Networks for Instructional Improvement in Jordan, 
2009-2012” and “Jordan School Learning Networks Project Interim Report, April 2013” 
can be found at http://www.cpre.org/jordan-report 
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A substantial part of the documentation work in this second phase concentrated 
on gathering data to establish a baseline understanding of program participants’ 
instructional practice. This was done through collecting a sample of videos of 
classroom instruction from the third cohort of teachers at the beginning of their 
participation in the program and through a baseline survey designed to measure 
participants’ initial attitudes and dispositions, values, and reported practices related 
to the big effects practices that have been at the core of the training. Subsequently, 
the research team collected the same types of measures toward the end of the 
program. These data allowed CPRE to estimate the extent to which the Project has 
effected change, particularly in the instructional core. In the subsequent sections, 
the findings are presented giving emphasis to Cohort 3 since they were a part of 
the major transitioning of the program during which more of the Project was 

led by QRTA staff. 


Data Sources and Methods 
This report relies on the following sources of data: 


= Participation records maintained by QRTA regarding attendance of teachers, 
principals, supervisors, and others at Project meetings and workshops; 


= Questionnaires from CPRE and QRTA trainers regarding their roles in 
implementing the project and perceptions of the transition of the leadership 
of the Project to the Jordanian counterparts; 


= Surveys of participants 


a Administrative records maintained by QRTA on the internal growth in staff 
members to support the Program; and 


m Videos collected from a sample of Cohort 3 teachers at the start of the 
Program and toward the end. 


THE SCHOOL NETWORK FRAMEWORK 


The School Learning Networks Project centers on creating a culture of continuous 
improvement in schools by providing long-term professional development support 
for teachers and principals through focused workshops, meetings with teachers 
across network schools, and school-site support for teachers. Through the Project, 
participating teachers attended a series of intensive, content-specific two-to-three- 
day workshops over the course of a few years. They also attended Network 
meetings, conducted at schools, designed to allow participants to learn from each 
other and share their successes and challenges in implementing some of the 
instructional strategies introduced during workshops. Table 1 (on page 12) lists the 
number of times that the various cohorts of educators had the opportunity to 
participate in workshops or meetings and the percentage of those invited who 
chose to attend. Project staff also visited participating schools to support use and 
refinement of workshop strategies through structured classroom observations, 
feedback and discussions with teacher teams. School principals and supervisors 
received training that provided them with tools and strategies to support and 
reinforce the instructional improvement efforts within their schools and to provide 
feedback to teachers as they implemented these new strategies. This training 
complemented the teacher professional development. The number of workshops 
varied both across cohorts and networks due to scheduling difficulties, but, on 
average, each group of participants had at least 7 workshop opportunities. 
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Table 1. Cohort Attendance for Project Activities: February 2011- September 2014 


Teacher Workshops 


Leadership Workshops 


Teacher Network Meetings 


Source: ORTA Workshop records 
Blank cells indicate workshop did not take place during the time period. 


Participation and Attrition 


Table 2. Program Participation Rates by Cohort 


Mathematics 


Science 


English Writing 


* “Others” includes Ministry of Education officials, CADER staff, winners of the Queen Rania Al Abdullah Award for 
Distinguished Teacher, and members of the design team. Source: QRTA workshop record 
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Across the three cohorts and subject networks, 2,158 teachers, 894 school leaders, 
and 104 other educators participated in the School Network Project, totaling 3,130 
participants over the almost 6 years of the Project’s implementation. A breakdown 
of the number of initial participants by Cohort and Network as well as the percent- 
age of participants who left the program can be found in Table 2 (on page 13). 


Each cohort targeted educators from different regions of Jordan (See Table 3 below). 
The educators in Cohort 1 were from Amman, which is also where QRTA’s 
headquarters are located. The networks for Cohort 2 were from governorates, or 
provinces, in the southern region of Jordan. Some of the schools were more than a 
two hour drive from Amman making it difficult for participants to come to QRTA, 
so the workshop teams often held the professional development sessions at 
network schools. The Cohort 3 governorates were slightly closer to Amman as 

they were in the Northern region of Jordan, but still quite a distance from Amman. 
Again, workshop leaders transferred much of the professional development to 
schools within the networks. 


Table 3. Cohort Locations 
by Governorates (Provinces) 


GOVERNORATE 


Cohort 2 Math Karak; Tafeileh 
Science Karak, Tafeileh 
English Aqaba 


Despite the extended and intensive nature of the Project that required participation 
over three years to complete the program, most teachers and school leaders 
sustained participation throughout the course of the program. Even as the cohorts 
grew and the locations of the networks became further from QRTA, the percentage 
of educators choosing to continue the program remained between 76% and 80%, 
on average (see Table 2 on page 13). Participants who missed two or more 
consecutive workshops were called by QRTA staff and asked if they planned to 
continue to participate in the program. CPRE counts those who confirmed that 
they would not continue the program as dropouts. Across the three cohorts, the 
dropout rates were highest in English writing. These higher attrition rates are likely 
due to the fact that the official ERfKE curriculum traditionally focuses very little on 
extended writing. Giving attention to writing through the Project requires taking 
time away from the ERfKE lessons. The writing workshop model also requires a 
significant shift in instructional practice that may be uncomfortable for some 
English writing teachers. Language also may have been a barrier for teachers since 
the instruction as well as the writing is in English, a second language for most 
teachers and students. 


Attendance rates at individual workshops were also relatively high (see Table 1 on 
page 12). The attendance rates were calculated as the percentage of invited 
workshop participants who actually attended the workshop. Workshop attendance 
rates ranged from 48% to almost 100%, with an average of 74% for both leadership 
and teacher workshops held during this time period. Attendance at the teacher 
Network meetings was slightly lower with an average of about 68% of teachers 
attending the meetings. Not surprisingly, attendance rates tended to be slightly 
lower at workshops and meetings later in the Program cycle. 
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Modifications and Adaptions to Professional Development 

As discussed in the prior documentation reports, CPRE and QRTA trainers made 
some significant changes following the first year of the professional development 
program. In year two, the teacher workshops and support for Cohort 2 and the 
second-year workshops in Cohort 1 were adapted in response to feedback about 
teachers’ needs, schedules, and school contexts, as well as the cultural context. 
For example, in mathematics and science, content was revised and condensed to 
allow for the reduction of the workshops from four-day to two-day sessions, in 
which half of the participants attended one of two two-day sessions to allow for 
more time for discussion among participants and personalized feedback from 
workshop leaders. In English writing, trainers modified workshop curriculum 
materials to better address the variation in students’ English-language skills, and 
adapted the language used in workshop materials to better match the teachers’ 
English-language abilities. Additionally, English writing trainers were able to 
illustrate the writing process using examples of work by Jordanian students—a new 
resource that some trainers believe has led to greater buy-in from participants 
earlier in the professional development cycle. Teacher workshops leaders across all 
networks adapted their materials to ensure that the instructional strategies and 
lessons could be used with minimal resources and in large classes. Modifications 
were also made for leadership workshops. In the second year of the Cohort 1 
leadership workshops, in response to limited implementation of workshop 
strategies inside schools and the perception that the work was not connecting 
directly enough with practice, workshop leaders began to hold workshop sessions 
in schools and initiated learning walks in schools with small groups of participating 
principals. Some of the trainings were also moved to the school site for later 
cohorts in the second and third years of the training. 


As Cohort 3 schools began the program, the content and focus of the professional 
development workshops stabilized and remained substantively the same as in 
Cohort 2. The focus of the work since 2011, which includes the final Cohort 2 
trainings and all of the Cohort 3 trainings, was on fine-tuning the workshops, 
rather than making any significant shifts in their content or organizational design. 
Reports from CPRE and QRTA trainers suggest that these minor adaptations and 
adjustments occurred constantly and were part of a continuous improvement 
process. Trainers relied on feedback from and discussions with teachers during the 
independent work time in the workshops to determine whether the teaching points 
of the session were clear to teachers. The training teams met each day after the 
workshops to review the work, and to make changes required to enhance 
engagement, understanding, and use. 


Based on workshops and school visits from prior cohorts, trainers identified some 
of the challenges teachers faced in implementing the instructional strategies 
presented in the workshops. With this information, they developed potential 
solutions or supports to assist implementation. Feedback from teachers and leaders 
during on-site support and Network meetings about challenges also informed 
workshop and on-site support adaptations. English writing trainers used student 
work that teachers brought to the workshops as evidence that workshops were 
effective or needed improvement. Feedback from complementary work, particularly 
the professional development that QRTA staff and CPRE trainers provided in 
Jerusalem, cross-fertilized with the Jordanian professional development to 
stimulate further improvements. 
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FINDINGS FROM CPRE’S THREE FOCUS AREAS 


Change: The Reflective Classroom Initiative 

The Reflective Classroom Initiative was a smaller effort within the Project that 
collected and analyzed video observations of teachers in the classroom setting. It 
began in 2012 with the third cohort of teachers in an effort to capture changes in 
teacher practices, over time, and to determine if these changes reflected the targeted 
practices that were presented and emphasized in the professional development 
workshop sessions. Additionally, the video observations allowed the research team 
to determine which areas of the program appeared to be most successful and those 
that were potential growth areas. 


The Study. Teachers were filmed in 2012 when they began the Network Project 
and then, again, in 2014 as they neared the end of the program. On average, the 
research team recorded 2 lessons for each teacher in 2012 and 2 lessons in 2014. 
Participation was voluntary and it was made clear to teachers that the focus of the 
Initiative was to learn more about how the Network Project was influencing 
participants’ classroom practices. While the goal was to measure change over time, 
the evaluation of teachers was not a part of the analysis. Both the identities of the 
teachers and students in the videos were kept confidential. 


Due to resource constraints, the research team could only videotape a sample of the 
Cohort 3 teachers. Researchers randomly sampled the Cohort 3 participants first by 
school and then within subgroups based on the subject the teacher taught and 
grade level in an effort to ensure the sample represented the program participants. 
Teachers within the sample were then asked to participate noting that participation 
was voluntary. Initially, 31 teachers agreed to be videotaped, but only 25 of the 
original participants completed the Initiative by participating in the videotaped 
observations in 2014. The Initiative yielded 86 videotaped lessons. Each video was 
coded by two trained coders, who were also fluent in Arabic. Coders used the 
Upgrade Observation Protocol”, which was adapted by CPRE researchers from the 


2 The Upgrade Observation Protocol can be found at http://www.cpre.org/jordan-report 


UTeach Observation Protocol (UTOP). The instrument can be used to assess the 
overall quality of classroom instruction in English/writing, math, and science from 
kindergarten to the undergraduate level. The Upgrade Protocol is composed of 

34 indicators across 4 major categories: Classroom Environment, Lesson Structure, 
Implementation, and Content. Each indicator was rated on a 5-point Likert Scale 
(1 to 5), with a N/A (Not Applicable) rating option for a few designated items. Most 
indicators asked the rater to assess the frequency of the practice as well as the 
quality of the practice. There were also 4 indicators that rated the teacher on the 
overall quality of each category. 


The indicators in each section specifically measured practices that align with the 
“big effects” practices from the Network Project. The Classroom Environment 
section assessed the degree to which the classroom environment was conducive to 
the learning of mathematics, English/writing and science, and how the teacher 
facilitated and created this setting. The Lesson Structure section assessed the 
organization of the lesson—opening, work period, and closing—and how well the 
teacher organized the sequence of learning activities during the class period, and 
the degree to which this organization facilitated the learning of English/writing, 
mathematics or science. The Implementation section assessed the instructional 
decisions, strategies, and practices that the teacher used during the lesson, how well 
the lesson flowed, and whether the teacher ensured that all students remained 
engaged in the content being covered. The Content section assessed the quality of 
the mathematics, English/writing, or science content being constructed by students 
during the class period. Although there were indicators within the Content section 
that measured the teacher’s content knowledge, it is an important to note that this 
section was meant to address the quality of the content that students were exposed 
to during class, both directly communicated by the teacher and learned through 
other means like discussion and independent practice. 


CHANGING CLASSROOM PRACTICE: The Evaluation of the School Network Project in Jordan 


Chart 1. 
Indicators where Teachers Demonstrated the Most Positive Change On Average 


a Negative HM No Change ™ Positive 


1.1 Frequency of collegial student group work 33% 4% 63% 


1.1 Quality of the collegial student work 33% 4% 63% 


1.5 Quality of the classroom organization 


to facilitate learning 25% 21% 54% 


2.4 Quality of resources used in lesson 25% 21% ey 


2.3 Quality of the lesson structure 


around engagement 25% 2% 58% 


3.2 Frequency that the teacher 


involved all students 21% 17% 63% 


3.2 Quality of the involvement of all students JRREYA 25% 63% 


3.3 Frequency of formative assessment 29% 4% 67% 


3.6 Quality of the implementation of safe 


and appropriate strategies 13% 50% SEES 


3.7 Frequency of student participation 


and engagement 42% 17% 42% 


4.1 Quality of content 


(appropriate, relevant, etc.) 29% 13% 58% 


0% 20% 40% 60% 80% 100% 


* Indicator Categories: 
Indicators numbered 1=Classroom Environment, 2=Lesson Structure, 3=Implementation, and 4=Content. 


Video Findings. The overall findings from the video observations are positive with 
the majority of teachers showing positive changes in their practice. On average, 
participants saw positive growth on over 80% of the indicators (27 of the 34) in 
the study, which measured the overall quality of classroom instruction (see Chart 1 
on page 20). Of the 27 indicators on which teachers demonstrated positive 
growth, more than half of the indicators with the greatest change were in two 
developmental areas, their command of the classroom environment and their 
lesson implementation. Examining specific indicators, the three indicators on 
which teachers demonstrated the biggest change measured the ways that the 
teacher engaged students. Sixty percent of the video participants improved 
anywhere between a quarter of a scale point to two and half points on these 
indicators. More than half of the 24 participants improved by between a half of 

a point to almost two points in their ratings for creating environments conducive 
for student engagement and collegial work. While not at the same magnitude, 
teachers also demonstrated positive change on all of the indicators for how they 
structured their lessons. 


Female teachers tended to have greater changes in their ratings between their initial 
videos at start of the program and the videos from the end of the program, but it 

is important to note that female participants from the Cohort 3 agreed to 
participate in the full Reflective Classroom Initiative at a rate two times that of 
male participants. Sixteen female participants completed the Initiative compared 

to only 8 male participants. 


When compared to the pre-program video observations, teachers demonstrated 
positive change in the frequency in which they used formative assessment in their 
lessons; however, the quality of the formative assessments and subsequent 
modifications within the lessons still needed improvement. These findings indicate 
that the teachers understand that formative assessment, a “big effects” practice, is 
important and are trying to use it in their classrooms. More time or likely, more 
training, is needed for them to implement it a way that is effective. 
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Self-reported Changes and Challenges from Surveys 


Table 4. Cohort 3 End-of-Program Survey Response Rates 


* Calculated as percent of Initial Participants. 


Table 5. Cohort 3 Table 6. Cohort 3 
Teacher Respondent Leader Respondent Background 


Source: Cohort 3 End-of-Program Leadership Survey 


Source: Cohort 3 End-of-Program Surveys 


Cohort 3 participants were surveyed? when they started the program and toward 
the end of the program. The survey included items about teacher and school leader 
background characteristics, use of various instructional practices and participation 
in communities of practice, and school context. Survey items were drawn from 
CPRE surveys that were conducted in prior years, as well as other resources, 
including items developed by the Bay Area School Reform Collaborative (2002) 
teacher survey, Consortium on Chicago School Research (2001) teacher survey. 
There were challenges with the administration of the end-of-program survey. 
Participants were assigned individual unique IDs when they completed the 
pre-program survey; however, these IDs were not used in the administration of 
the end-of-program survey limiting the researchers’ ability to link participants 
responses across the two surveys. For this reason, the majority of the survey 
analysis will focus on participants’ responses as they completed the Project. 
Response rates for the Cohort 3 End-of-Program Surveys participants are displayed 
in Table 4 on page 22, and background of respondents are displayed in Table 5 
and Table 6 on page 22. 


Response rates varied across the networks with the Math Network having the 
highest number of survey participants and school leaders having the lowest rate of 
response for Cohort 3. Response rates for the Science Network were also lower than 
preferred. The surveys were administered in the workshops toward the end of the 
program when attendance was lower. The surveys were also made available online, 
but not all of the teachers have access to the Internet making the workshop setting 
the only way for these teachers to access the survey. 


Given the variation in participation, the results of these surveys should be 
interpreted with some caution. Specifically, it is possible, but unlikely, that 
participants who chose to complete the surveys are also those participants who 

are particularly engaged in the program, so the findings regarding change and 
frequency of use of instructional practices may be higher than the total population 
of participants as a whole. While the researcher team collected program completion 
survey data from Cohort 2 participants, an in-depth analysis is not included in 

this report since participation rates were below 50% for multiple networks 

in this cohort. 


3 The End-of-Program surveys for each Cohort 3 Network can be found at 
http://www.cpre.org/jordan-report 
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Teacher Findings. The findings indicate that the improvement in classroom 
practices among the video participants was similar to the changes reported by their 
peers in Cohort 3 on the surveys. The majority of teachers in Cohort 3, across the 
networks, reported making a change in how often they used student collaboration 
and engagement in the classroom, the two areas in which the largest positive 
changes were observed in the video study. Nearly all of the science teachers (91%) 
responded that they had increased how often they encouraged student discussion 
in class and 86% of both English writing and math teachers noted the same 
changes. More than 80% of the math and English writing teachers reported 
increases in their use of mixed groups or pairs to allow students at different levels 
to support each other. Over 70% of English writing teachers also reported positive 
changes in this type of student collaboration. 


Overall, the majority of the survey participants across the networks noted positive 
changes in their practice. On average, more than 80% of the surveyed teachers 
said that their was an increase in the amount of time for student reflection on 
their learning and the processes of the lesson, teacher use of questioning to assess 
learning and adjust their lessons, as well as their use of more student centered 
practices. 


Challenges. Similar to the shared strengths and noticed changes in practice from 
the video study, the challenges reported by the video participants were also 
challenges for the teachers surveyed. Teachers’ content knowledge was the area in 
which teachers demonstrated the least change between the pre-program videos and 
the videotaping toward the end of the program. This finding is not surprising 

since school leaders, support staff, workshop team members and even teachers 
themselves noted that content knowledge could be a weak area for some teachers. 
More than 60% of English and math teachers surveyed cited their knowledge of 
their content area as a limiting factor for them in the classroom. Slightly fewer 
science teachers cited this reason, but still almost half noted it as a challenge. 
Almost 90% of school leaders sited teachers’ knowledge of their content area as a 
limiting factor. While content knowledge was not a major focus of the program, 
increased understanding of the big ideas in the curriculum was expected to be a 
by-product of participation. The supports and workshop sessions tended to focus on 
the “big effects” practices and used the critical content as a medium to demonstrate 
how the practices could be applied to the subject. Additional support and a 
potential knowledge-building component in specific content areas may be a 
necessary addition for future programs with similar teachers. 


Time for collaboration with one’s colleagues was a major challenge for Cohort 3 
teachers across the networks, but teachers appear to be meeting more frequently 
than when they started the program. Fewer teachers reported “never” meeting 
with colleagues to discuss their instruction. The percent of teachers surveyed who 
reported “never” meeting with their peers to discuss content specific instruction 
decreased by an average of 15% for math and science respondents but only 2% for 
English teachers. This change indicates that more teachers were meeting with 
colleagues more frequently. More teachers also reported meeting on a daily or 
weekly basis, with the exception of English/Writing teachers. The percentage of 
science teachers reporting collaborating regularly more than doubled going from 
22% to 54%, math and English/writing teachers reported mixed changes. When 
considering the amount of time set aside for the meetings, roughly 60% to 70% of 
all surveyed teachers said that they were only able to meet for a few minutes or 
about a half an hour whenever they did meet. These reports are only marginally 
different from the amount of time that teachers reported having when they began 
the Network program meaning that, while teachers are able to collaborate more 
often, the amount time spent in these meetings is still limited. 


Although more teachers reported meeting daily or weekly with colleagues who 
teach the same subjects, the goal is for a large majority if not all teachers to meet at 
least on a weekly basis. With the exception of the science network, which had 
slightly more than half of teachers collaborating regularly, only about a third of 
teachers are meeting either daily or weekly. That means that 70% of those surveyed 
are meeting only a few times a semester if at all. 


Leadership findings. On average, 80% of the school leaders who responded to the 
surveys noticed changes in teachers practices related to the goals of the Project as 
well as changes in their own practice and the level of encouragement given to 
teachers who were trying to implement the program practices. Most of the school 
leaders said that there was an increase in the number of times that supervisors and 


CHANGING CLASSROOM PRACTICE: The Evaluation of the School Network Project in Jordan 


principals observed teachers. The majority of principals and supervisors also saw a 
notable change in the types of questions that participating teachers were employing 
in the classroom, noting that they were higher level questions and that teachers 
were probing for student understanding of content. School leaders also noted 
increases in their level of encouragement of student discussion and in their 
observation of teachers’ use of student discussion in lessons, 79% and 87% 
respectively. Eighty-five percent of the leadership participants noticed a change in 
the level of encouragement they were giving to teachers for the use of group work 
in the classroom. It is worth noting that less than two-thirds of principals and 
supervisors noticed an increase in “the amount of time that teachers have during 
the regular school week to work with colleagues on curriculum and teaching 
methods.” While 64% still includes the majority of the respondents, it is much 
lower than the change reported in the other areas explored. Interestingly, it is 
reflective of the challenges reported, both through survey responses and 
anecdotally, by teachers, school leaders, and workshop team members. Each group 
reported that finding time to collaborate with peers was an ongoing challenge for 
teachers. More than 70% of school leaders surveyed cited the lack of time available 
for planning and lesson collaboration as a limiting factor for teachers. 


Concerns about teacher’s content knowledge and instructional abilities also were 
expressed by the Cohort 3 principals and supervisors who participated in the 
end-of-program survey. About half of the school leaders lacked confidence in their 
teachers’ ability to help struggling students become proficient. Even though the 
majority noticed an increase in teachers’ use of higher lever questions, almost 60% 
of the leadership respondent still questioned teachers’ ability to use the questions to 
promote student learning. Almost half of the principals and supervisors also lacked 
confidence in teachers’ ability to identify student misconceptions and address the 
conceptual errors. 


Similar Findings from Cohort 1 and Cohort 2 

The initial documentation report submitted in 2011 described and summarized 
the first-year implementation activities, including the professional development 
program, content, and participation rates. The findings suggested that there were 
high participation rates in both workshops and network meetings, and moderate 
rates of attrition, with slightly higher rates of attrition in the writing network than 
in the other networks attributed largely to the lack of focus on writing in the 
Jordanian curriculum as well as the time demands associated with the use of the 
writing workshop. Workshop team leaders worked with the ministry to make it 
clear to English writing teachers that it was acceptable to use the workshop 
practices in their classrooms. The workshop team noted that this support helped to 
some degree with teacher buy-in and attrition. The first-year report also examined 
participants’ response to the Project, including satisfaction with the program as 
well as attitudes toward and self-reports of the use of the strategies introduced and 
supported through the professional development program. Overall, teacher and 
principal views of the workshop were positive regarding the organizational aspects, 
leadership of the workshops, and content of the workshops. Participants reported 
that much of it was new material that was relevant, and that it was useful in 

their classroom. 


Building Capacity within QRTA 

When the Partnership began the Project, one intention was to build the 
organizational capacity of the QRTA so that it could provide high quality, effective 
professional development. Two particular aspects of building that capacity were 
developing staff and establishing quality assurance mechanisms. Developing 
QRTA staff would enable them to lead and support instructional improvement in 
participating schools. This effort included expanding QRTA’s staff through a 
targeted recruitment strategy that sought educators with appropriate content 
knowledge, pedagogical content knowledge, and familiarity with Jordanian 
public schools, and an understanding of the core instructional concepts and 
practices advanced by the Partnership. 
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Since 2009, the staff contributing to the Project has almost tripled. In 2009, there 
were only 9 team members with the majority being at the management level. 
Today, there are 25 team members supporting the project including workshop team 
leaders and facilitators as well as a researcher leading future measurement and 
evaluation of the Project. CPRE team members used an apprenticeship model to 
prepare current QRTA staff to design and lead professional development. The 
training and transfer was gradual with QRTA staff taking on greater responsibility 
for the workshops with Cohort 3 (See Table 7 below). After each workshop, 
participants were asked to provide anonymous survey feedback. This feedback was 
used to get educators’ perspectives on what was presented and to adjust future 
workshops if necessary. This feedback was also used to track the quality of the 
transition and continuity of the content presented. The feedback data were 
analyzed from all three cohorts to assess whether changes in participants’ feedback 
occurred over time as QRTA workshop leaders took on a greater role in leading the 
workshops. The feedback on the workshops’ organization, content presented, and 
usefulness remained pretty constant, on average, across the cohorts (See Table 8 on 
page 29) indicating that QRTA workshop team leaders were delivering professional 
development that was on par with that of the CPRE workshop leaders. 


Table 7. Transitioning of Responsibilities to ORTA Staff Developers 


COHORT 1 NETWORKS COHORT 2 NETWORKS COHORT 3 NETWORKS 
TC/CU Partner Role Lead in Content Co-deliver Advise as 
and Delivery Workshop Sessions ORTA Leads 


Source: Growing School Networks for Instructional Improvement in Jordan, 2009-2010, 2011. 


Table 8. Average Teacher Views on the Organization and Facilitation of Network Workshops: 
November 2010-September 2014 


“The workshop was 
100.0% 98.7% 92.9% 


well-organized.” 


92.5% 96.8% 


"The leaders of this 
workshop were 
knowledgeable about 
[my content area] 
math/science/writing.” 


VAN Ddi% Dd,1% 


Source: QRTA Workshop Feedback November 2010 to September 2012 


As previously noted, workshop attendance varied throughout each cohort and 
network (See Table 1 on page 12). Some workshops had greater attendance than 
others. Workshops with lower attendance meant that fewer program participants 
were able to engage in that professional development opportunity and fewer were 
able to provide feedback on the experience. It is important to note that those who 
did attend the workshops did not always opt to complete the anonymous surveys, 
so the feedback on the workshops may not be representative of all those who 
attended (See Table 9 and Table 10 on page 30). 
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Table 9. Cohort 2 Workshop Feedback Surveys Response Rates 


Source: QRTA Workshop Feedback April 2012 to September 2014 


Table 10. Cohort 3 Workshop Feedback Surveys Response Rates 


Source: ORTA Workshop Feedback April 2012 to September 2014 


Sustainability 

Building the capacity of QRTA staff was a part of a larger goal for the Project, 
ensuring that the program and its practices were sustainable within the schools. 
Part of ensuring sustainability is making sure that required supports for 
implementation are in place. Following the first year of the project, on-site support 
at schools was provided somewhat unevenly by QRTA staff. On-site support for 
Cohort 1 teachers averaged close to two visits per year. Over the second cohort’s 
first two years of participation, schools were visited an average of two times in the 
mathematics network, once or twice in the science network (a mean of 1.4 visits per 
school) and once in the English writing network with Cohort 3 visits following a 
similar pattern. The long travel distance between English Writing Cohort 2 schools 
in Aqaba as well as among the Cohort 3 schools in Mafraq, Irbid, Jarash and Ajloun 
and the QRTA offices in Amman likely contributed to the lower number of on-site 
support visits for those networks. Nonetheless, across all Networks, the goal of 
providing 12 hours of on-site support for each school annually (approximately 
three school visits per year) was not met. QRTA leadership and staff also provided 
limited, if any, on-site support for principals through school visits. Geography 
proved to be a challenge for the first three cohorts and could impede the Project 
from sustaining reforms within schools beyond the workshops and even beyond 
the program as participants were not fully supported as planned while they had 
access to the Project. The QRTA staff are aware of the importance of on-site support 
and are making efforts to meet participants needs. 


The teacher meetings are another component of the Project that support 
participants. The goal is for each cohort to hold four Network meetings per year, 
which initially was only partially met for Cohort 1 since only the Science and 
English Writing Networks had four or more meetings. A similar trend continued 
with Cohort 2 and 3 where the English writing network held fewer than four 
teacher meetings, but both the Science and Math Networks held four or more 
teacher meetings. The increase in meetings held, with some networks holding 
more than four, suggest that the QRTA staff were able to address some of the 
obstacles to implementing frequent network meetings making it more likely that 
the practices of the Project were implemented by the school communities. 
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Cohorts 4 and 5. While the CPRE team has not had a major role in working with 
the additional cohorts, they are aware that QRTA began work with a fourth and 
fifth cohorts. Adding these cohorts to the project means that more public school 
teachers in Jordan will have access to the supports and practices emphasized in 
the Project. Increasing educators’ exposure to the Project increases the number of 
educators who have had extensive program training and will allow for more 
collaboration with other participants from previous cohorts further expanding the 
program network. Additionally, these participants can serve as leaders within their 
schools sharing their knowledge and practice with colleagues and helping to create 
communities of practice. A development that program teams already have seen 
happen within the other cohorts where teachers and school leaders report that 
they are sharing their knowledge with the peers and encouraging them to use the 
practices. More data on how the instructional improvement efforts are sustained 
within schools that participated in the Project will likely come from additional 
follow-up with participants from prior Cohorts now that they have completed 

the program. 


LESSONS LEARNED 


Surveys and Data. The end-of-program surveys were administered to all 
participants, regardless of whether they completed the program. Researchers at 
QRTA wanted to include all participants since it was possible that they may have 
benefited from the program even though they did not complete all of the program 
components. While having these data may provide some insight, it is possible that 
the program had a different impact on participants who only partially engaged in 
the program when compared to their peers who were able to fully engage in all 
aspects of the program until its completion. Future evaluations should attempt to 
capture the experience of partial participants, but solicit their perspective separate 
from participants who complete the program. 


The effectiveness of a survey is largely dependent on the researcher’s ability to use 
language to convey the ideas to the participant. In the case of the Project, the 
surveys were created in English and then translated to Arabic. This process meant 
that, at times, the nuances of the words chosen in English got lost or did not quite 
convey the same ideas once translated. This challenge was one that was noticed 
earlier in the project so individuals who spoke both languages were brought in to 
help better convey these nuances in future surveys. Though it still was not perfect, 
it did improve the overall quality and understand of what was being measured. 


In general, data collection presented challenges especially since much of the initial 
responsibility was placed on QRTA, which was a newer and growing organization at 
the beginning of the project. As a result, little baseline data were collected from the 
first two cohorts. It is possible that greater data collection could have taken place 
with more oversight from CPRE researchers. It might have also been beneficial for 
CPRE to manage the initial data collection since it was the more experienced 
organization. Then, the process could have been transitioned to the QRTA 
researchers much in the way that the workshop facilitation was transitioned. 


Capacity Building. QRTA is a relatively new organization and has great aspirations 
for the work they can do as an organization. In an effort to realize some of its goals 
and obtain additional funding to sustain the organization, staff sometimes were 
charged with multiple tasks related to more than one initiative, which often 
divided their attention. There was never any question about the QRTA team 
members’ dedication to the project or their belief in the work, but, at times, the 
needs of other initiatives left team members with less time for the Network Project. 
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The unfortunate consequence was that some components, like site visits, were not 
fully implemented in the manner intended. When engaging in similar work in the 
future, it will be important to focus on not only building capacity in the form of 
additional staff members but also on ensuring that those gaining the knowledge 
and expertise needed to lead the program are able to devote the needed time to the 
project so that they are able to fully engage in that work. 


Language Barriers And Gender Considerations. The CPRE trainers’ lack of 
proficiency in Arabic has posed one of the more significant challenges in imple- 
menting the program. Providing feedback when Jordanian trainers were presenting 
was often difficult because concurrent word-for-word translations do not usually 
allow time for nuanced interpretation, a challenge given the importance of 
specificity and language in determining levels of conceptual understanding. 
Determining whether to press participants about specificity or clarity during a 
discussion of a concept was challenging since CPRE trainers were not always able 
to identify whether the problem was in the translation, or if the problem was the 
actual participants’ understanding. One important adjustment was continually 
refining the language used during training and in the training materials as 
concepts become clearer to the Jordanian trainers and their understanding 
deepened. Additionally, one of the benefits to having QRTA trainers take on more 
responsibility for training was that less time was spent during the trainings 
providing translation. The transitioning of the project to local education trainers 
enabled more in-depth discussion on the concepts and practices that the 

Project emphasizes. 


Gender was another area where the Project had to be thoughtful. Workshop leaders 
as well as other program team members noted teachers’ level of engagement and, 
at times, their professional knowledge varied. Female teachers tended to have 
stronger credentials and be more open to growing as a professional, on average. 
Their attendance and effort was greater than their male counterparts. There were 
exceptions with some male participants displaying all of these characteristics and 
very eagerly engaging in the program and, vice versa, there were female participants 
who were quite the opposite. Team members knew that there might be gender 
differences based on anecdotes about how different the all-female schools are from 
the all-male schools. Greater attention should be given to anticipating these 
differences and incorporating approaches to managing them for participants. 


CONCLUSION 


Overall, the School Network Project has had a positive impact on both educators 
and the staff at the Queen Rania Teacher Academy. 


1. Teachers are reporting that they are using the “highly effective” practices that 
were emphasized in the workshops. This increased use was also noticed in 
the video observations of the teachers who participated in the Reflective 
Classroom Initiative. 


2. School leaders are seeing positive changes in teachers’ efforts to engage students 
and assess their learning as well as changes in other areas. 


3. School leaders also reported changes in their own practices noting that they 
were observing teachers more frequently and that they were more supportive of 
teachers using the “big effects” practices. 


4. CPRE team members were able to support and train QRTA team members to 
lead the workshops and other initiatives related to the Project. 


5. QRTA has grown as an organization and has almost three times as many 
staff members to support instructional support in the years to come. 


Continuing to engage the program participants and support them in their schools 
is an on-going effort. Teachers emphasize a need for a stronger professional 
community that collaborates regularly. QRTA has acknowledged the challenges 
that the partnership faced during the first five years of implementation and is 
exploring ways to improve the program. Adapting to the needs of the participants 
was a critical part of the program and continuing to assess the needs of educators 
and how best to meet those needs likely will continue to be a key component 

to the program’s success. 
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The partnership between CPRE and QRTA formed through the facilitation of 
CUMERC with the goal of improving the quality of the public education system 
through the provision of high quality in-service training to current teachers. 
Through this partnership, three cohorts of teachers have spent extensive amounts 
of time deeply engaging in researched-based practices and taking this gained 
knowledge back to their schools and classrooms. As the Project continues to grow, 
the professional community of Jordanian public school teachers will expand 

and it is the hope of the partnership that this community of educators will 

be able to meet the needs of their students and prepare them for the competitive 
world that awaits them. 
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